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Proposed work 


This was a continuation proposal to test and implement a novel multiple model 
framework for nonlinear plants whose dynamics are rapidly varying in a state space that 
is assumed known. During this research we had the opportunity to compare in the same 
dataset two parallel approaches to implement the gating function in the multiple model 
framework. We also had the opportunity to evaluate a new and intriguing nonlinear 
dynamical system approach called the echo state network (ESN) to implement adaptive 
critics. Unfortunately, we had to stop at the system identification stage, because a no cost 
extension request was not granted. 


Productivity 

During the time of this grant we published three journal papers and one conference 
proceedings, acknowledging the NASA support: 

Cho J., Principe P., Erdogmus D. and Motter M., "Quasi-Sliding Mode Control Strategy 
Based on Multiple-Linear Models," NeuroComputing , Volume 70, Issues 4-6, January 
2007, Pages 960-974 

Cho J., Principe P., Erdogmus D. and Motter M., "Modeling and Inverse Controller 
Design for an Unmanned Aerial Vehicle Based on the Self-Organizing Map," IEEE 
Transactions on Neural Networks . Volume 17, Issue 2, March 2006 Page(s):445 - 460. 

Lan J., Cho J., Erdogmus D., Principe J., Motter M., Xu J., "Local Linear PID Controllers 
for Nonlinear Control", Int. J. of Control and Intelligent. Systems, Special Issue 
Nonlinear Adaptive PID Controls, vol 33, #1, 26-35, 2005 

Xu D., Lan J. Principe J., “Direct Adaptive Control: An Echo State Network and Genetic 
Algorithm Approach” IEEE Int. Joint Conf. on Neural Networks (IJCNN05) , Montreal, 
Canada. 


Summary of the findings 

We compared two possible implementations of local linear models for control: 
one approach is based on a self-organizing map (SOM) to cluster the dynamics followed 
by a set of linear models operating at each cluster. Therefore the gating function is hard 
(a single local model will represent the regional dynamics). This simplifies the controller 
design since there is a one to one mapping between controllers and local models. The 
second approach uses a soft gate using a probabilistic framework based on a Gaussian 
Mixture Model (also called a dynamic mixture of experts). In this approach several 
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models may be active at a given time, we can expect a smaller number of models, but the 
controller design is more involved, with potentially better noise rejection characteristics. 

Our experiments showed that the SOM provides overall best performance in high 
SNRs, but the performance degrades faster than with the GMM for the same noise 
conditions. The SOM approach required about an order of magnitude more models than 
the GMM, so in terms of implementation cost, the GMM is preferable. The design of the 
SOM is straight forward, while the design of the GMM controllers, although still 
reasonable, is more involved and needs more care in the selection of the parameters. 
Either one of these locally linear approaches outperform global nonlinear controllers 
based on neural networks, such as the time delay neural network (TDNN). Therefore, in 
essence the local model approach warrants practical implementations. In order to call the 
attention of the control community for this design methodology we extended successfully 
the multiple model approach to PID controllers (still today the most widely used control 
scheme in the industry), and wrote a paper on this subject. 

The echo state network (ESN) is a recurrent neural network with the special 
characteristics that only the output parameters are trained. The recurrent connections are 
preset according to the problem domain and are fixed. In a nutshell, the states of the 
“reservoir” of recurrent processing elements implement a projection space, where the 
desired response is optimally projected. This architecture trades training efficiency by a 
large increase in the dimension of the recurrent layer. However, the power of the 
recurrent neural networks can be brought to bear on practical difficult problems. Our goal 
was to implement an adaptive critic architecture implementing Bellman’s approach to 
optimal control. However, we could only characterize the ESN performance as a critic in 
value function evaluation, which is just one of the pieces of the overall adaptive critic 
controller. The results were very convincing, and the simplicity of the implementation 
was unparalleled. 


Result Highlights 

SOM-Based Competitive LLM 

Suppose that input-output data pairs of the form {(u\,y\),...,(uN,y N )}, where a is 
the input signal and y is the output signal, is available from a SISO system for system 
identification. It is assumed that the system that generated this data sequence is of the 
following general nonlinear time-invariant form (assuming that there is no instantaneous 
effect of the input on the output): 

X A+1 ~ f k) /-j\ 

yk= h (*k ) 

In the state space, this nonlinear dynamical equation can be approximated by an ordinary 
linear differential equation of the fonn: 

yk = a \yk-\ +•••+ a n yk-n +b\ u k-l +- + b m u k-m (2) 

The coefficient vectors a and b depend on the local regime; consequently system 
identification using multiple local linear models of this form necessitate the quantization 


3 



of the state space. The quantization can be adaptively achieved by training a SOM on the 
combined state vector x defined as 

X A- = [v/t-l ••• yk-n u k-\ ••• Uk-mY ( 3 ) 

The SOM consists of a multidimensional array of neurons with weight vectors w„ which 
are trained competitively on the input vector samples x*. While the samples are presented 
to the SOM one at a time, in a series of epochs, the weight vector of the winner neuron 
w w for the particular sample (i.e. the weight vector that minimizes the Mahalanobis 
distance d(w,x k ) = (w-x yt ) r 2 _ 1 (w-x / -.) between the two vectors) and its topological 
neighbors w„ are updated according to the following rules [13]. 
w w ((+i)<-w w (()+#i:" 1 (x t -w w ( 0 ) 
w„(t + ])<-w n (t) + ri(t)- (4) 

A(l|w„( 0 -w w ( 0 |].,cr( 0 )i: _ 1 (x t -w„( 0 ) 

In (4), the (Gaussian) neighborhood function h(.,o) is annealed in time such that initially 
the neighborhood radius o covers most of the network and towards the end of training it 
is narrow enough to include only the winner neuron. Typically linear or exponential 
annealing schemes are used for both the neighborhood radius and the step size 77 . 

Once the SOM is trained, the available data is partitioned into smaller sets; the 
samples associated with weight vector w, are {(x a ,y n ),...,(x iN .,y iN .)} , where /V, is the 

number of samples clustered to neuron i. Each neuron also has a linear models attached to 
each it, whose coefficient vectors a, and b, can be optimized using least squares with this 
input-output training data that is clustered to this neuron in the SOM. The overall LLM 
output can be produced either using hard competition or soft competition between the 
linear models associated with each neuron. In particular, the model output is determined 
as follows: 

M 

n = X Pik ^ b ^ ]x *' ( 5 ) 

i = 1 

For hard competition, the scaling factor p ik is either 1 or 0 depending on the 7 th neuron 
being the winner or not for input x k . For soft competition, which also allows a smoother 
transition between the linear models at switching boundaries, the model output can be 
determined by a weighted average. In this case, one can use, for example, a weighting 
based on the Mahalanobis distance again: 

Pik =d(y/ i ,\ k )f^ j A j t =l d(w J,x k ) ( 6 ) 

Alternatively, similar weighting schemes where only a few of the nearest models are 
activated could be employed. From a probabilistic point-of-view, such a weighting 
mechanism can be regarded as the probability of the corresponding linear model being 
responsible for the sample under consideration. The GMM approach described in the 
following section makes explicit use of this probabilistic view for the data. 
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Figure la. (left panel) Schematic diagram of the SOM-based LLM approach for system identification. The winner neuron 
is illustrated by the full circle. 

Figure lb. (right panel) Schematic diagram of the GMM-based LLM approach for system identification. 


GMM-Based Cooperative LLM 

Suppose that input-output data pairs of the fonn {(ui,yi),...,(uN,yN)} are available 
from a SISO system as described in (1). The linear approximation in (2) still holds locally. 
The representative state vector is again described as in (3). In the GMM approach, instead 
of partitioning this vector into disjoint sets using a SOM, it is assumed that the data is 
generated by a multi-dimensional joint mixture of Gaussian distribution. In other words, 
the distribution of vector x is given by 

M 

P( x k) = ^ J a i G ( x k’l l i’' L i) ( 7 ) 

;=i 

where G(x;p , £) is a multivariate Gaussian density with mean p and covariance Z. The 
coefficient a, denotes the probability of occurrence of this mode in the GMM. Given the 
training data, once the vector samples x/ c are constructed, the maximum likelihood 
solution for the parameters p„ and S, can be determined using the expectation 
maximization (EM) algorithm [14]. Alternative statistical model selection approaches 
could also be employed. As in the SOM-based model, here it is assumed that a linear 
model is associated with each of the Gaussian model in the GMM. Each linear model has 
weight vectors a, and b, that can be optimized using least squares once the parameters of 
the GMM are determined using the EM algorithm. The overall model output is a 
weighted average of individual linear models 

M 

h=^Pidaf b f]*k= QT Zk ( 8 ) 

i = 1 

where p ik =a i G(\ k ;\i i , E f ) . The overall linear model weight vector is defined as 
0 = [af bf •■•a ^ b J M \ , and the corresponding overall modified input vector is 
z k =\P\k x l ••• P\ik x l I • The linear models can be optimized simultaneously using least 

squares considering the available output measurements va as desired outputs 
corresponding to the modified input sample z*. Fig 1 compares the block diagrams fo3 
the two implementations for a graphical display of the differences. 
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Figure 2a. Schematic diagram of the switching linear PID controllers based on the SOM decision for the winner model. 
Figure 2b. Schematic diagram of the weighted average PID controller based on the GMM assessment of contributions. 

Local Linear PID Controller Design 

To illustrate how to develop controllers for these architectures, the PIDs will be 
utilized since they are so widely used. Notice that here a globally nonlinear control action 
can be implemented, from the set of local linear models. The literature has an abundance 
of PID design methodologies for linear SISO systems including direct pole-placement 
techniques and optimal coefficient adjustment according to some criteria [24,25]. In the 
case of MIMO systems, where the system identification phase is carried out as described 
in the previous sections with the necessary extensions and modifications to the SOM and 
GMM approach, one can resort to MIMO PID design techniques [26]. 

Given the local linear models as obtained through the use of a SOM or a GMM, 
and a PID design technique, the overall closed loop nonlinear PID design reduces to 
determining the coefficients of the individual local linear PID controllers using their 
respective linear plant model transfer functions. For both SOM-based LLM and GMM- 
based LLM methodologies, one needs to determine a set of PID coefficients per linear 
model. In the competitive SOM approach, the model output depends only on a single 
linear model at a given time; therefore, the PID coefficients are set to those values 
determined for the instantaneous winner model (as illustrated in Fig. 2a). In the GMM 
approach, since the models are averaged with probabilistic weights, the same must be 
done with the PID controllers. An alternative look at (8) shows that the GMM-based 
LLM output is also equivalently expressed as 

( m r. l) r- 1 

T v d i T “ik ( Q\ 

,i=, ‘ V g, (9) 

G=i 1 1 lkJ 

where a ik and b ik denote the weighted average linear model at time instant k. 
Consecutively, The instantaneous PID controller will be characterized by a weighted 
average of the PID coefficients of all linear models as follows: 

PID k =TZi p * PID ‘ (10) 

where PID, is the transfer function of the PID controller for the I th local model (as 
illustrated in Fig. 2b). In the case of a soft combination SOM-based LLM as described by 
(6), the formulation in (10) can be employed for the PID controller. 

Simulations 
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Figure 3a. Response of the closed loop nonlinear PID control system to smoothly varying desired response is shown at 
the top for the SOM-based LLM approach. The bottom plot shows the index of the winner linear model changing in 
time as determined by the SOM based on the instantaneous input. 

Figure 3b. Response of the closed loop nonlinear PID control system to various step changes in the desired response is 
shown at the top for the GMM-based LLM approach. The bottom plot shows the contribution coefficients of the linear 
models changing in time as determined by the Gaussian mixture model based on the instantaneous input. 


The performance of the proposed local linear PID design approach will be 
demonstrated on a simplified SISO nonlinear missile dynamic model where only the yaw 
dynamics are considered [5], The simplified two-state dynamics are described by: 

•5 C 

i] = x 2 -0.1 cos(jci)(5jci - 4x l + x, )-0.5 cos(.x: 1 )m 

i 2 = — 65 jcj +50.VJ 3 -15xj 5 -x 2 100w (11) 

y = *i 

Since the input is the rudder deflection, it is limited by ±0.5 radians in practice. The 
simulations invoke this limit on the control input value although the PID design will not 
take this into consideration. 

Two models were identified using 6000 samples of input-output pairs obtained from 
the dynamical equation (11) using a discretization time step of T s =0.05s (which 
corresponds to 300s of flight time). In order to excite a rich variety of dynamical modes 
in the plant, the system identification data was generated using a uniformly random input 
signal within the specified limits. The embedding delays for the input and the output were 
both selected to be 2, resulting in 4-coefficient linear models. This embedding delay was 
intuitively chosen in accordance with the dimensionality of the state dynamics, but was 
also tested experimentally. Increasing the embedding delay did not result in system 
identification performance improvement. The SOM-based LLM approach utilized a 
15x15 rectangular grid of neurons, amounting to 225 competitive local linear models, 
whereas the GMM-based LLM approach assumed to 5-mode mixture model, resulting in 
5 cooperative linear models. The identified models were also tested on original 50s- 
length data (1000 samples), generated using a new sequence of random input. The actual 
plant output, the model predicted output and the estimation error for the SOM-based 
LLM and GMM-based LLM models are provided in Fig. 3a and Fig. 3b. The signal-to- 
error ratio (SER) for these two models on the testing set are found to be 3 1 ,7dB and 3 ldB, 
respectively. 

The PID controller design is carried out using the standard pole-placement technique 
[24]. For both modeling methodologies and all linear models, the corresponding PID 
coefficients are determined to bring the closed-loop response poles from the plant output 
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Figure 4a. Response of the closed loop nonlinear PID control system to various step changes in the desired response is 
shown at the top for the SOM-based LLM approach. The bottom plot shows the index of the winner linear model 
changing in time as determined by the SOM based on the instantaneous input. 

Figure 4b. Response of the closed loop nonlinear PID control system to various step changes in the desired response is 
shown at the top for the GMM-based LLM approach. The bottom plot shows the contribution coefficients of the linear 
models changing in time as determined by the Gaussian mixture model based on the instantaneous input. 


to the desired output to 0, 0, 0.05+/0.3, and 0. 05-/0. 3. This closed-loop nonlinear PID 
controller is tested in two cases: one with various step changes to the desired output (Fig. 
4a and Fig. 4b). 

The step and tracking performance of the designed nonlinear PID control scheme in 
closed-loop operation with the actual plant model is very satisfactory when compared 
with the performance of an adaptive nonlinear time-delay neural network (TDNN, not 
shown). However, there are differences in terms of the switching, where the GMM is 
achieving the same control law by switching very fast (chattering) among the models 
during the transitions, while the SOM achieves a smooth transition. We hope this 
example illustrates well the plus and minus of each control architecture with controllers 
that are well known in the community. 


System ID based on ESN model. 


Echo State Networks (ESN) first proposed by Jaeger are a new kind of recurrent 
networks with simplified learning mechanisms which largely decrease training time. The 
state model of an ESN can be written as 


\S k+l =f(W in .X + W.S k )- 

\Y k+ i=K» t -s k+1 


( 12 ) 


where W in is input weights, W is internal weights, W out is output weights, S represent the 
internal states, X and Y are corresponding to system input and output. The ESN has a 
large “reservoir” recurrent network that can produce diversified representations of an 
input signal, which can then be instantaneously combined into an optimal manner to 
approximate a desired response. As a competitor of time delay embedding methods (like 
the delay line used in both the SOM and GMM), ESN captures the dynamics in the 
recurrent connections. The input vector is connected to a “reservoir” of N discrete-time 
recurrent networks by a connection matrix Wi„. At any time instant k, the readout S k (state 
output) from the RNN reservoir is a column vector. The reservoir states are transformed 
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by a static linear mapper. Each processing element (PE) in the reservoir can be 
implemented as a leaky integrator and the state output or the readout is given by the 
difference equation in (12). 



System ID through ESN 

The big advantage of the ESN is that only the output weights are trained, which 
represent a training problem linear in the weights, for which even analytic techniques 
(least squares) are available. Therefore, the training becomes much easier and also much 
fewer samples are required to accomplish it. 

Figure 6 compares the system identification performance on the Lorenz system 
and the missile system, and illustrates well that the ESN is capable of making a good 
state representation even without using time embedding. The prediction performance is 
better for the ESN than that from time-embedding data (on an average of around 3dB). 
For the LoFlyte system, since the performance from LLM has been over 40 dB, there is 
no 3dB advantage from ESN (SER from ESN is 40.8379 dB), yet the input dimension for 
ESN is decreased from eight (one step embedding) or 12 (two steps embedding) to four 
(only current state). The training time is dramatically decreased. The promising aspect of 
the dimension decreasing and fast training is not only reflected on the simplification of 
algorithm, it will save time and resource for on-line control as well. 
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Fig. 6, Left: Prediction performance from GMM-LLM (two-step embedding); 
Right: Prediction perfonnance from ESN (zero embedding). 


Desire and prediction. 



(a) 
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Single readout. 




(b) 

Fig. 7, (a): Prediction performance from GMM-LLM for the LoFlyte with embedding; 
(b): Prediction performance from ESN for the LoFlyte without embedding. 
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